ETH Library EDGE MATCHING AND 3D ROAD RECONSTRUCTION USING KNOWLEDGE-BASED METHODS

Road network extraction from aerial images has received attention in photogrammetry and computer vision for decades. We present a concept for road network reconstruction from aerial images using knowledge-based image analysis. In contrast to other approaches, the proposed approach uses multiple cues about the object existence, employs existing knowledge, rules and models, and treats each road subclass differently to increase success rate and reliability of the results. Finding 3D edges on the road and especially the road borders is a crucial component of our procedure and is the focus of this paper. We developed an algorithm for automatically matching edge segments across images. The algorithm exploits edge geometric and photometric attributes and edge geometric structures. A framework to integrate these information sources using probability relaxation is developed and implemented to deliver locally consistent matches. Results of straight edge matching are presented and future work is discussed. Prologue I first encountered the name Wrobel in some publications related to image correlation when I was doing my Master Thesis at OSU at the beginning of the 80s. After coming to ETHZ in 1984, and since my Ph.D. topic was on matching, a research field where Prof. Wrobel had many parallel activities, we had many opportunities to meet and discuss. 1985 the workshop on matching in Darmstadt, December 1985 watching together the winter swimmers in Cannes, 1987 jogging (not me) in Interlaken, and of course the many ISPRS events and some conventions of DGPF. Thus, it came as no surprise that Prof. Wrobel was finally the co-referent for my Ph.D. I remember that he read my lengthy text with great attention, making very concrete proposals on what to change, and being prompt in his reactions even under very difficult circumstances with his family. But what impressed me more, was the fact that in my overview of matching methods, I made several critical remarks on the object-based matching method that he co-developed, and later refined under the name FAST. He respected my opinion, we discussed and cleared-up several aspects, but at no stage did he try to „press“ me to make only positive comments on „his“ algorithm. In August 91, I was on holidays with my wife and one of my brothers in Sardinia. I had to make corrections to my Ph.D., deliver a new version, and have my examination (on a Friday the 13!) two weeks after the holidays’ end. Thus, I took some work with me, including Prof. Wrobel’s comments to study them. I know it’s a rare thing in Italy, but it did happen to us. Our car was broken in the middle of the day at a beach full of people and everything was stolen. We could only contact the local police next morning. Finally, at around 11.30 the police office opened. An officer, il colonelo X, was typing very seriously and very slowly at an old typewriter his report. At the end he asked whether we had lost any things of particular value. I stated a few personal things, including Prof. Wrobel’s remarks, not hoping for anything. In the afternoon, we got a phone call to go to the police station. All things of exceptional value had been found! Presumably thrown in a field. Prof. Wrobel’s notes did not have a scratch. They were still placed neatly in a plastic enclosure as I had put them, all in the proper sequence. I was happy, and my brother was swearing because he had forgotten to name some exceptional value things, which he could have received back. This is the story of Prof. Wrobel’s corrections, which were respected even by the robbers. Apart from our personal relations, Prof. Wrobel had always had close ties to our group at ETHZ. He was twice with us as guest professor giving lectures, and helping young researchers, was co-referent of Ph.D. dissertations and sent us a group of students for a common Praktikum. I always admired his serious and helpful comments, his consistent and persistent research work, digging deep into the problems, and not just scratching the surface or jumping here and there on „modern“ topics. His is definitely not the type of show-man, but his contributions in photogrammetry, for us and the coming generations, are significant not just because of his scientific output on important topics, like matching, but also attitude-wise. His paradigm was: you don’t have to do many things simultaneously, but those you do, do them well! We wish Prof. Wrobel and his family health, spiritual satisfaction and happiness in his „Ruhe“stand. As contribution to his Festschrift, the Ph.D. student Chunsun Zhang and me would like to present some research on edge-matching, a topic that is related and complimentary to Prof. Wrobel’s work on his area-based matching approach, and 3D road reconstruction. Introduction The extraction of road networks from 2D aerial images is one of the current challenges in digital photogrammetry. Due to the complexity of such images, their interpretation for mapping roads and buildings has been shown to be an extremely difficult task to automate (Gruen et al., 1995, 1997; Förstner and Plümer, 1997). As digital topographic databases have been created in many developed countries, and now need to be updated, several methods are explored to incorporate this existing knowledge for image interpretation. The effect of this incorporation is twofold: the existing information provides a rough model of the scene, that will help the automation process, while the old database gets revised and updated with the latest aerial images. The here presented work is part of the project ATOMI. ATOMI is a co-operation between the Federal Office of Topography (L+T) and ETH Zurich. Its aim is to use aerial images and automated procedures to improve vector data (road centerlines, buildings) from digitised 1:25,000 topographic maps by fitting it to the real landscape, updating it, improving the planimetric accuracy to 1m and providing height information with 1-2m accuracy. In the current tests, we use 1:16,000 scale colour imagery, with 30 cm focal length and 25% sidelap, scanned with 14 and 28 microns at a Zeiss SCAI. The input data are: the nationwide DTM (DHM25, 25 m grid spacing and accuracy of 2-3/5-7 m in lowland/Alps), the vectorised map data (VEC25), the raster map with its 6 different layers and the digital images. The VEC25 data have an RMS error of ca. 5-7.5 m and a maximum one of ca. 12.5 m, including generalisation. They are topologically correct, but due to their partly automated extraction, some errors might exist. The general overview of ATOMI is described in Eidenbenz et al. (2000). In this paper we deal with road reconstruction, while the building part can be found in Niederöst (2000). In a first step of road reconstruction, we aim at detecting existing roads, while roads that do not exist anymore or new ones will be treated later. We first concentrate on roads, ignoring other transportation network objects like railway lines, mountain paths etc. In this paper, we present the ongoing work of the project on the interpretation of aerial images. The ultimate goal of this work is to build an automatic and robust system to reconstruct the road network from aerial images with the aid of an existing road database and possibly additional data. In order to increase the success rate and the reliability of the results the system will contain a set of image processing tools, and make full use of available information as much as possible. At the first stage, we will focus on the design of methodologies and techniques for image analysis tools, in particular we are interested in using information extracted from old database and applying rules and models to guide the process. Finding 3D edges on the road and especially the road borders is a crucial component of our procedure and is the focus of this paper. Straight edges are extracted by fitting the detected edge pixels in each image and straight edge matching across images is performed through exploiting rich edge attributes and edge geometrical structures. General strategy of road network extraction A large number of approaches for road extraction have been proposed and published. The strategies fall into two broad categories. Semi-automatic schemes require an operator to provide interactively some information to control the extraction. In McKeown and Denlinger (1988), Vosselman and de Gunst (1997), Airault and Jamet (1996), the extraction algorithms are based on road tracking starting from an initial point and direction, extraction of parallel edges by extrapolation and matching of road profiles. With a few points of a road segment provided by an operator, (Gruen and Li, 1995, 1997) developed a LSB-Snake method to extract roads simultaneously in multiple images. This is advantageous because the solution is more constrained and the result is more robust, particularly in areas which are occluded in only some of the images. These semi-automatic approaches can be extended to fully automatic operation by means of automatic seed point detection (Zlotnick and Carnine, 1993). The common automatic methods first extract reliable hypotheses for road segments through line and edge detection and then establish connections between road segments to form road networks (Bajcsy and Tavakoli, 1976; Fischler et al., 1981; Ton et al., 1989; Wang et al.,1992; Trinder et al., 1999). The contextual information was taken into account to guide the extraction of roads in Ruskone (1996). In Baumgartner et al. (1997) and Mayer et al. (1997), road detection is based on the extraction of lines in an image of reduced resolution through scale-space analysis (Steger, 1998). Baumgartner then extracted edges in the original image, where the final result was the combination of results at two resolutions based on a set of rules, while Mayer adopted ribbon snakes to verify roads and discriminate them from other line-type objects in the original images by means of width consistency. Knowledge-based methods involve the use of existing GIS databases or maps and rulebased systems. Stilla (1995) and Bordes et al. (1997) used maps and cartographic databases respectively as a guide for image interpretation. In Vosselman and de Gunst (1997), the old database is used not only to verify it but also to detect new road branches from the given data. In ATOMI, we develop a new approach for automatic extraction of 3D road network from aerial images which integrates knowledge processing of colour image data and existing digital geographic databases. The information of existing road database provides a rough model of the scene. Colour aerial images give the current situation of the scene, but are complex to analyse without the aid of other auxiliary data. Therefore, the information provided by the existing geographic database can help the understanding of the scene, while the images provide real data useful for improving the old road database and updating it. The system under development strongly relies on the following three aspects: • Use and fusion of multiple cues about the object existence and of existing information sources. All cues have associated relevant attributes. • Use of existing knowledge, "rules" and models. The road model includes geometric, radiometric, topological and contextual attributes. • Object-oriented approach in multiple object layers (hierarchical division of classes in subclasses, division of a class according to terrain relief and landcover). The initial database is established by the information extracted from existing geographic data and road design rules. This offers a geometric, topological and contextual description of road network in the scene. The database is automatically updated and refined using information gained from image analysis. Colour cues, expressed in the form of colour region attributes, are also used to support stereo matching and improve the performance of 2D and 3D grouping when combined with geometric cues. Since neither 2D nor 3D procedures alone are sufficient to solve the problem of road extraction, we propose to extract the road network with the mutual interaction of 2D and 3D procedures. Hence, the main steps of road extraction are: building up of the knowledge base for each road segment in VEC25, finding 3D straight edges in a search region defined by the VEC25 data, classification of image patches, extraction of other cues, combination of various cues guided by the knowledge database to find plausible groups of road edges for each VEC25 road segment and refinement and update of the knowledge database. The general strategy is shown in Fig. 1. Fig. 2 shows more details of the results of image processing and derivation of subclass vector attributes. Figure 1. Strategy of road network extraction in ATOMI Straight edge matching 3D edge generation is a crucial component of our procedure. We are interested in 3D straight edges because they are prominent in most man-made environments, and usually correspond to objects of interest in images, such as buildings and road segments. They can be detected and tracked relatively easily in image data, and they provide a great deal of information about the structure of the scene. Additionally, since edge features have more support than point features, they can be localised more accurately. The 3D information of straight edges is determined from the correspondences of edge segments between two images. Due to the complexity of aerial images, different view angles and occlusions, straight edge matching is a difficult task. Existing approaches to edge matching in the literature are generally categorised into two types. One is directly Input data from L+T Road design rules, other knowledge Database for road network Class Road type Road marks Geometry with 3D info Width Length Horizontal & vertical curvature Topology info Landcover Image processing Results Accuracy estimation Partial results (Fig.3) Cue combination Controller Knowledge Base comparing the attributes of a edge in one image with those of a set of edges in another image and selecting the best candidate based on a similarity measure (McIntosh and Mutch, 1988; Medioni and Nevatia, 1985; Greenfeld and Schenk, 1989; Zhang, 1994). The similarity measure is a comparison of edge attributes, such as orientation, length, edge support region information etc. In another strategy, the edge correspondence is found by performing structural matching. Structural matching seeks to find the mapping between two structural descriptions. A structural description consists of not only features but also geometrical and topological information among features. A number of methods have been developed for structural matching (Vosselman, 1992; Haralick and Shapiro, 1993; Christmas et al., 1995; Cho, 1996; Wilson and Hancock, 1997). Figure 2. Details of image processing and derivation of subclass vector attributes The developed method in this paper exploits rich edge attributes and edge geometric structure information (Fig. 3). The rich edge attributes include the geometrical description of the edge and the photometric information in the regions right beside the edge (flanking regions). The epipolar constraint is applied to reduce the search space. The similarity measure for an edge pair is first computed by comparing the edge attributes. The similarity measure is used as prior information in structural matching. The locally consistent matching is achieved through structural matching with probability relaxation. The details of the method are described below. Figure 3. Flow-chart of straight edge matching Edge extraction The input images are first filtered with Wallis filter for contrast enhancement and radiometric equalisation (Baltsavias, 1991). The technique developed in a previous project (AMOBE) is used to extract straight edges (Henricsson, 1996). The edge pixels are detected with the Canny operator. An edgel aggregation method is applied to generate contours with small gaps bridged based on the criteria of proximity and collinearity. All segments are checked using their direction along their length and split at points where the change in direction exceeds a given value. A test is conducted to see whether the consecutive segments can be merged into a single straight edge. For each straight edge segment, we compute the position, length, orientation, and photometric and chromatic robust statistics in the left and right flanking regions. The photometric and chromatic properties are estimated from the “L”, "a" and "b" channels after an RGB to Lab colour space conversion and include the median and the scatter matrix. Input Image